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AESTRACT 

Plans for evaluation should be laid prior to 
implementation ot a program. A hierarchy of types of program 
evaluation helps select the exact type of evaluation intended. This 
hierarchy starts with the easiest (and least useful) type of 
evaluation and six progressively more difficult levels of evaluation 
are added in cumulative fashion. The seventh and final type is a 
master plan for evaluation^ which should be accomplished by starting 
with the top steps and their interrelationships and working down. The 
evaluation types^ with applicable questions for extension leader 
orientation^ are: (1) inputs made — How much time and money are 
expended in leader orientation^ and is this the right amount? (2) 
activities performed — What kinds of orientation activities are new 
leaders engaged in^ and are there the right number of activities? ^3) 
recipients involved — With how many leaders do we allocate certain 
amounts of orientation time and activity? (4) reactions — How do 
volunteer leaders react to their orientation? (5) "KAS" change 
{change in knowledge^ attitudes^ and skills) — Which development are a 
consequence of orientation? (6) "practice" change — Do the attitudes^ 
skills, and knowledge gained through orientation help the volunteer 
in working with a-H participants? (7) results achieved — Do the 
volunteer roles help achieve ultimate aims with a-H youth? Evaluation 
at the lower levels is meaningless without answers to higher 
evaluative questions. (KM) 
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A HIfHAM:HY OF PROGRAM iJVALUATION 
APPLIED TO VOUJNTEJiR LEADERSHIP DEVELOPMBNT 

For Extension program personnel, "evaluation ' Is usually the last thing 
that comes to mind: last because most abstract formulations of the prograri 
development process place "evaluation" as the final step In a multi-phase 
sequence. Last in another sense, too; evaluation is often neglected Iji 
allocation of time and funds, in favor of "getting the Job dene." Is it 
possible that tne frequent neglect of evaluation may be due partly to zhe 
conceptual approach mentioned, which places evaluation as the last phase? 

If evaluation is usually an afterthought, planners of this workshop are to 
be commended In providing a tlme^or state teams to make plans and assignments 
for an actual evaluation "back home." I am convinced that evaluation vill 
never take its rightful place in education, including Extension, until plans 
for evaluation are speclficaUy laid prior to iJiiplementatioD of the educational 
program. 

KliroS OF EVALUATKMI 

Altho'ifih we are Just beginning the "evaluation section of tne workshop, " 
it cannot be denied that a ma^or part of our formal activity so far has been 
purposely evaluative. 

In the broad sense of the tenn, "evaluation" is assigning a degree of 
\'alue to something. Identifying a problem concemlug volunteer participation, 
then, is a type of evaluation; some situation is defined as bad, or in need of 
ljuprovement. 



In the same way, state team have selected one or more possible or probable 
solutions for each problem. This, too, is a type of evaluation, as sane 
potential solutions have been tentatively assigned more value than ethers. 

And now the focus of the workshop turns to selecting an evaluation task 
for planning here, and for carrying out at home. What is suggested lov is for 
state teams to plan for a more systematic evaluation— more tovard the 
"scientific research*' end of the continuum than you have done so far. 

Perhaps a framework for selection of an evaluation task would assist our 
thinking at this point. Since any set of ''pigeOTiholes'* is sonevhat arbitrary, 
this framework raey not be the best, but it will serve to focus group dis- 
cussion, and hopefully improve conmunications. 

We can raise at least three kinds of evaluative questions in lilx^ension: 
First, "situational evaluation." This type of evaluation 
basically defines problems. 

Second, "process evaluation." This is an evaluation of 

program planning, recruitaaent, admlnlstratiCHi and utilization of 

knowledge gained from educational progranmiing. 

Third, "program evaluation." For example, we could ask how 

much o U-H volunteer program does to improve a situation. This 

question implies at least a bench mark or longitudinal study. 

Thus, you may wish to plan for collection of further data to checK 
whether you have correctly defined the problems you have already identified as 
state teams. Or, you coulJ plan an evaluation of a new extension process, 
such as how roles are allocated to personnel. Or, thirdly, you may wish to 
plan for collection of data to assess the results of a new program. 



Frutchey (1957) has defined evaluation as (l) collecting infonnation, ^l- 
a basis for oaKlnc a decision or Judgment; (2) applying certain standards or 
criteria to thlo .olle-rted infomiation, and (3) forming the Judgment or making 
the d..clj:ion. Uslne Frutchey's definition of evaluation, the tesk you heve at 
hand includes : 

1. Deciding what Information to collect about a situation, n 
process, or a procram's results. 

2. Deciding to what criteria you will relate this information; 
i.e., ascertaining the general end specific objectives. 

3- Collecting the needed information and applying it to said 
criteria or objectives. 

A HIERARCHY OF PROGflAM fiVAIAJATION 
Frutchey's is one of the most concise definitions of evaluation I have 
come across. But being very abstract, it does not tell us what types of infor- 
raation to consider. I would now like to present another facet of the frame- 
work for selection of a specific program evaluation task. This is a hierarchy 
of types of progran evaluation. The hierarchy serves to help select exactly 
what type of evaluation is Intended (c.f., Catalanello and Klrkpatrick, 1963). 

Let me try to explain the hierarchy in general terms (see page 12). 
Type I in the hierarchy at the left indicates the easiest type of evaluati.on 
but unfortunately, also the least useful; the hierarchy proceeds to the right, 
through more and more difficult and complex types of evaluation, vhJch are at 
the same time more and more useful. Tyi^ VII In the hierarchy i^. a master 
plan for evaluation. Administrators, program leaders, and evaluntors can 
pick and choose from among these types of evaluation, basing their choice upon 
the Gi7.e of the project envisioned, the resources allocated for cvolualion, 



and competciicy of tr.e evaluation staff. Certainly this typolog;/ iz an cver- 
simpllTication, but the advantage of a model is to simplify reality so that v. 
can gee some kind of e "mind-hold" on it. 

Six levels of evaluation are added in cumulative fashion as we ^- > from 
left to right. Notice how a neans-end hierarchy (Suchnan, I9OJ: 52-p9) ^ is 
built Into this hierarchy of evaluation. If we are to accomplish anything, we 
must expend some effort (input). Not any kind of effort, but educationel 
activities , and we must reach an intended audience (reciplT?nts) . But if the 
recipients don't n:)preciatc the activities (reactions) they won't uc Motivated 
to Icam anythin.j ("':< A S" change). If their learning ic toward educational 
objectives, do they apply this learning ("practice" change)? And are they 
really better off (results achieved) if they follow practices we re-ormcnded? 
Thus we climb the hierarchy in order to reach the goals of the Coopcr&tive 
Extension Service. 

Let's take a detailed look at eacn type of evaluation beginning vith trie 
simplest tyyye of evaluation, Type I, or "inputs made." This is a "reporting 
type" of evaluation which indicates how much time and money is beinc spent in 
the program. A Type 1 evaluation may also indicate how many staff are 
employed, their educational characteristics^ etc. Is this really evaluation? 
Mostly it is just description, but it is also evaluation if we assume at 
least some correlation between how much effort is spent and how much is 
accomplished. (We arc certainly willing to assume this correlation in adding 
personnel to a program.) But more Impoi^nt, we also need "inputs made*' data 
for cost-benefit analysis within the "higher types" of evaluation to be 
discussed below. 
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A somewhat better evaluation is Type TI, which besides tcUin, us nov 
much effort is expended also describes by n.eans of written and audio-vinual 
accounts the kindn a:.d frequencies of activities perfonned in the project. 
This type of evaluation generally assumes that the activities held lead to a 
positive result. For example, who would deny that taking inner-city children 
on a field trip to a fam is beneficial'; The trouble with thic type of 
assumption is that it may not be tnie in certain situations, and moreover the 
question remains as to the extent of the benefits obtained. Of .ourcc, one 
variable in the extent of benefits is tne number of recipients of tne progran./ 
end their degree of involvement. Therefore, we generally employ a TyTe III 
evaluation to cauje not only the number of those cMentele or r.-clpier.ts who 
are directly contacted, but also those who are reached indirectly by mass media 
and by message diffusion. Again, it is cocnon to assume that the more 
recipients, the better the program, perhaps even qualitatively. Of course, it 
18 important that the audience have the "right" social characteristics, e.g., 
disadvantaged, urban, etc.. In tenns of the objectives of the program. 

So far, we have discussed types of data which are considered as objectively 
measurable; i.e., they are relatively ea.y to count. These data may not by 
themselves tell us a great deal, but they are comparatively easy to aether. 

Often, we wish to add at least a fourth level of evaluation, 'reactions," 
in order to seek confirmation of the hunch that given activities ere helpful. 
A Typo IV evaluation adds to the previously discussed types of data ■reactions- 
type data, or what people sa^ about the activities. Such data should be repre- 
.".cntative of the audience, and also be quantitative, so that intensity of "pro" 
or "con" reaction is measured. The audience, the staff of the project, or 
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panels of observers arc asked: "What l.i your attitude tovard tr.-- proici.t?" 
"Do you approve of it or disapprove of it? Why?" Valuable as such data may 
be, it may not indicate progress toward educational objectives. So, we niay 
wish to press on to evaluation Type V, "change in knowledge, attitudes and 
Skills." 

Type V evaluations provide data to ascertain whether the activities are 
simply entertalninc, or whether tney really bring about the development of 
the group or individual, in tems of the educational objectives of the project. 
Type V evaluation.-, show with quantitative data (a) whether chanccG have 
occurred in specific attitudes, knowledcn and skills, (b) the direction of 
such change, and (c) the distance or extent of change. Data is usually 
collected within teaching-leamlJig, or tost situations. But the knowledge, 
skills and attitudes to be acquired are usually considered in "practical" 
education es Just steppingstones to chancina a behavior pattern. 

Concern with quantitative data on acquisition of desirable patterns of 
behavior brings us tc Type VI, the "practice change" or adoption typo of 
evaluation. Continuous practice of the h-h-S-2 nutrition rule, or of certain 
safety measures are but two examples. Of general interest In adoption studies 
is whether the behavioral change is really innovation, or Just a modification 
of what is common practice. Duration of any change made is also of geicral 
interest. 

Finally, we reach the "results achieved" or the Typo VII evaluation, 
includine "ultimate aims." These "ultimate ains" generally include concepts 
of iiersonal and social development such as self-conf ider.ce, uonccrn for the 
group, etc. This hi(::hest type of evaluation, VII, permits c cost-bo; cfi . 
ratio to be estimated, through coroparinr qua; titative data on "rosuits achieved," 
with "inputs made." 



Although the lower types of evaluation may not have good mesBures on 
benefits received through the program, and thus have no way to compare such 
benefits with the cost of the project, every evalviation eventually has to 
answer the question, "Considering the total resources required, is the 
present program recomnended for further work toward the soluticm of the 
problem selected for attack? If not, what approach is suggested?" So whether 
or not there is valid, quantitative data to support a cost-benefit ratio 
analysis, this final Jud^nt regarding a program must be n«de. 

DBSCESDIMG IHK HIEaftRCHy 

We go up the "steps" to reach Extension goale, but we must cone down the 
steps In order to evaluate Extension programs. That Is, effective evaluation 
depends upon starting with the top steps and their Interrelationships, and 
logically, working our way down. Why? Because the principle to observe is 
this (Suchnan, I967) : you can't evaluate change at "steps" I through VI 
without knowing or assuming the relationship between change at the "step" in 
question, and all the "steps" above it in the hierarchy! 

Climbing the steps to reach Extension's goals is pretty clear. But the 
necessity of descending the steps In evaluation nay require examples, which are 
provided In an application of the hierarchy In the next section of this paper. 
To demonstrate the necesity of descending the steps in evaluation, I shall try 
Instead to climb the steps. 

EVAUJATION TYPES APPLIED TO LEADER ORIENTATION 

To exemplify the use of the hierarchy, I would like to use Dolan's (1969) 

phases of leadership development as a point of departure^ Let's take Dolan's 

• the phases are: leader identification, leader selection, leader orientaticn, 
leader training, leader utilization, leader recognition, leader evaluation. 



phase three, "Orientation of Leaders," and see how this hierarchy can be used 
la selecting an evaluation task. In the auestions raised below. Extension orient- 
ation iz evaluated, rather than the vol^^i.teer leaders. Also, only the added 
higher "level" of evaliiatlon of each successive type of evaluation is discussed. 
Type I. How much tlae and money do we expend In orieatation of 
new leaders , and is this the right amount of resources 

aM money) to expend? We can't answer whether the 
amount of input is correct without going to a higher 
type of evaluation, unless we know or are willing to 
asBume how effective our orientation activities are. 
Type II. In what kinds of orientation activities do we engage new 
leaders , and are we having the correct number of these 
activities ? Again, we can't answer this latter 
qiaestion without assuming or knowing how effective 
the orientation activities are.** 
Type III. With how many leaders do we allocate certain amounts 
of orientation tlae and activity ? What are their 
characteristics , and are the proper number of such 
• leaders receiving our orientation activities ? Once 
again, an answer to the last question restB on the known 
or assumed <|uality of the orientation activities.** 
Type IV. How do volunteer leaders react to their orientation 

activities ? Do they think they get enough orientation? 
tbQ'iBUCh? Notice that the answer to this question heXpa 



** Of course, there are additional criteria needed to fully answer this question, 
such as measures of efficiency and degree to which performance is adequate to 
the total amount of need (Suchatftn, 196?; 63). 
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answer the three preceding questions. For example, 
suppose ve accept reaction data as a valid evaluation. 
If it is the case that volunteera say ve are giving 
too nrach orientation, then the more such activities a 
leader is exposed to, the more leaders exposed, and 
the more time spent In providing orientation, the 
leas effective our program is J 
Type V. What developments in the volunteers ' knowledge . 

attitudes and skills are a consequence of our orlenta- 
^iPP activities? Ag^iln as with "reactions" data, 
"K A S change" data allows us to evaluate our "Inputs 
made," "activities held," and "contacts nade." But 
notice that we need "practice change" and "results 
achieved" data in order to evaluate "K A S change." 
We can measure whether "K A S change" occurs without 
knowing whether the skills, knowledge and attitudes 
learned are really the proper skiUs, knowledge, and 
attitudes . 

Type VI. Dc the attitudes, skills and knowled< ;e fiidned through 
orientatioD help the volunteer to assist and interact 
in recomnended ways with U-H participants ? As stated 
before, only when we have an answer to this question 
can we begin to evaluate changes In the volunteers' 
knowledge, skills and attitudes, brought about by the 
orientation program. 
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Typ; m. Do the prescribed volunteer roles help achieve ultimate 
aijns with gmth, e^., a better self -oopcept . 
h^r citizenship aspirations, etc. ? Again, it Is only 
as we answer this "ultlnate" question that we can 
evaluate the changes in patterns of behavior (pmctices) 
brought about by the orientation prograa (via changed 
skills, attitudes and knowledge). 

COMCUUSIOMS AHD IMPLICATIOIB FCR 
A STRATBCT OF EVALUATICW 

Any evaluative question within the hierarchy is « laced to, and has Hapll- 
cations for, all the rest of the questions. 

Evaluation at the lowest levels of the hierarchy is rather rueaningless 
unxess we either know, or make assumptions about the answers to "higher" 
evaluative questions. 

For example, it is meaiingless to ask "Are w^ reaching enough 
volunteer leaders with our orientation progrtun?" if we have no idea of: 

- How positively nev volunteers react tc the program. 

- Whether knowledges, attitudes and .ski] is gained 
through orientation help the voiunt««i3 perfonn their 
expected role. 

If a higher level evalu«tive question is answered pcsitively through 
research (e.g., the knowledge and ski^l gained orientation actually do 
help the volunteer assist in recoBi»ended ways), then the achievement of 
lower-l.^vel objectives :an be Interpreted as progress (although the 
progress may not be sufficient to meet th.» entr re necf ) . 

For exaniple, we coold conclude that if a greater proportion of our 
volunteer leaders are exposed to our orient* ti on prognm, the more our 
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leaders vlll, In fi&ct, assist in reconended mys. 

Therefore, Extension should stress evaluatiTS sr-v»le 't the higher levels. 
These evaluation studies can help provide a ba.. . .>r interpretation of 
less costly, lower types of evaluation. 
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